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ABSTRACT 

We propose a reweighted zero attracting least absolute devi- 
ation algorithm. The reweighted li norm penalty on the fil- 
ter coefficients is incorporated in the least mean absolute de- 
viation (LAD) algorithm to improve the performance of the 
LAD algorithm. The performance of LAD, zero-attracting 
LAD (ZA-LAD) and reweighted zero attracting least absolute 
deviation (RZA-LAD) are evaluated for linear time varying 
system identification under the non-Gaussian (a-stable) noise 
environments. Effectiveness of the RZA-LAD algorithm is 
demonstrated through computer simulations. 

Index Terms — LAD, ZA-LAD, RZA-LAD, sparse sys- 
tem identification, li norm, a-stable noise 

1. INTRODUCTION 

Adaptive filters have been widely used in channel equal- 
ization, echo cancellation, noise cancellation, linear predic- 
tion and system identification. Least mean squares (LMS), 
recursive least squares (RLS) and their variations are the 
widely used adaptive algorithms. Compared to recursive least 
squares (RLS) algorithm, the LMS algorithm has a slower 
convergence speed, however the implementation of the LMS 
algorithm also is less complicated than the RLS algorithm. 

In many applications, the unknown system response can 
be assumed to be sparse, only a small fraction of the coeffi- 
cients are different from zero. Zero-attracting filters exploit- 
ing the sparse nature of the system can improve the perfor- 
mance of adaptive filters. Motivated by LASSO [1] and recent 
progress in compressive sensing [2|, the zero-attracting LMS 
(ZA-LMS) algorithm and reweighted zero-attracting LMS 
(RZA-LMS) were proposed in |3|, for sparse system identi- 
fication. ZA-LMS is a combination of LMS algorithm with 
Zi-norm penalty of the coefficient vector. The reweighted li 
minimization algorithm is first proposed in [4] to enhance the 
sparsity of the system. 

The LMS type algorithms generally provide a more accu- 
rate solution, less misadjustment when the noise is Gaussian, 
however, LMS is very sensitive to the presence of outliers. 
Although the convergence rate of the least mean absolute de- 
viation type algorithms is low, they are robust to the presence 



of outliers, such as the a-stable noise. The a-stable noise 
model is a more generalized model [5 6], such that the Gaus- 
sian model can be seen as an special case of a-stable model 
by setting the characteristic exponent a = 2. 

In this paper, the performance of least mean absolute de- 
viation (LAD) [5 1, zero-attracting LAD (ZA-LAD) [7| and 
reweighted zero-attracting LAD (RZA-LAD) are evaluated 
for linear time varying system identification under the non- 
Gaussian (a-stable) noise environments. The coefficients are 
updated using fixed step size. 

The rest of the paper is organized as follows. The RZA- 
LAD algorithm is given in Section II. In Section III, we 
evaluate the performance of the three algorithms for linear 
time varying system identification under Gaussian and non- 
Gaussian noise environments. Conclusions of our work and 
some further research directions are provided in Section IV. 

Notations: Matrices and vectors are denoted by boldface 
upper case letters and boldface lower case letters, respec- 
tively. x(n) is the system input signal at time n, w(n) is the 
coefficient vector and d{n) is the desire signal at time n, The 
superscripts (■)^ denotes the transpose and sgn(-) denotes the 
sign function. | ■ | and 1 1 • 1 1 1 denote the absolute value and 
1 1 norm, respectively. The mth element of a column vector is 
denoted by [ J^. TZ represents the set of all real numbers. 

2. THE RZA-LAD ALGORITHM 

2.1. Review of LAD and ZA-ALD algorithms 

The output of the LAD algorithm is given by 

yi(n) = wf (n)x(n). (1) 
The cost function is formulated as 

Ji(wi(n)) = \d{n) - = |ei(n)| , (2) 

where ei(n) — d{n) — yi{n) is the error signal. 

Using the stochastic gradient approach, the filter weights 
are estimated iteratively by 

I ( \ dJi{wi{n)) 

wi(n + 1) = wi(n) - /xi — — — , (3) 

aw in 



where 



aji(wi(n)) 



-sgn(ei(n))x(n). 



(4) 



9wi(n) 

Substituting (|4|l into (O, we obtain the updating equation for 
the LAD algorithm: 



Wi(n + 1) = wi(n) + ^isgn(ei(n))x(n), 



(5) 



where fii is the step size that should be chosen carefully to 
ensure convergence. 

The output of the ZA-LAD algorithm is 



y2{n) = w^(n)x(n) 



(6) 



For the ZA-LAD algorithm, li norm penalty is used to explore 
the sparse nature of the filter coefficients. The cost function 
is formulated as 

J2(w2(n)) = \d{n) - y2in)\ + 02 ||w2(n)||j 

= |e2(")| + "2 ||w2(n)||;^ , (7) 

where 62 (n) — d{n) — y2{n) is the error signal, a2 is regu- 
larization parameter. 

Using the stochastic gradient approach, the filter weights 
are estimated iteratively by 



W2(n + 1) = W2(n) - ^2 



gj2(w2(n)) 
9w2(n) 



(8) 



where 

5J2(w2(n)) 
9w2(n) 



-sgn(e2(n))x(n) + Q;2Sgn(w2(n)). (9) 



where 63(71) = d[n) — yz{n) is the error signal, 03 is the 
regularization parameter, £3 is a positive number and M is 
the number of elements in weight vector W3(n). 

Using the stochastic gradient approach, the filter weights 
are estimated iteratively by 

W3(n + 1) = W3(n) - ^3 — ^ , ^ , (13) 



and 



dJs{w3{n)) 



-sgn(e3(n))x(n) +a3e3g(n). (14) 



where g(n) e TZ^^^^ and the mth element of g(n) is given 
by 



[sin)], 



sgn([w3(?i)],„ 
1 + esliwsin)] 



l<m< M. 



(15) 



Substituting (|9]l into we obtain the updating equation for 
the ZA-LAD algorithm: 

W2(n + 1) = W2(n) + fi2Sgn[e2{n))x{n) - r2Sgn(w2(n)) , 

(10) 

where /i2 is the stepsize and r2 = fJ.2Ct2- 

2.2. The RZA-LAD algorithm 

The output of the RZA-LAD algorithm is 

Vain) = wj{n)x{n). (11) 

Compared with li norm, log-sum penalty function is a 
better approximation for the sparsity of the filter coefficients. 
For the RZA-LAD algorithm, log-sum penalty function is 
used as the coefficient penalty function. The cost function is 
formulated as 

M 

^3(w3(n)) = -y3{n)\ +03 ^ log (l + £3] [w3(7i)],„ |) 

m— 1 

M 

= |e3(n)| +"3 X! log(l + '^3|[w3(n)],„|), (12) 



m— 1 



Substituting ( fT4l) into ( fT3] l, we obtain the updating equa- 
tion for the RZA-LAD algorithm. The updating equation is 
given by 

W3(n + 1) = W3(n) + /i3Sgn(e3(n))x(n) - r^gin), (16) 

where /i3 is the stepsize and — /X3a3e3 and g(n) is given 
indB]!. 

3. PARAMETER SELECTION A BRIEF DISCUSSION 

The ZA-LAD and RZA-LAD algorithms are regularization 
based adaptive algorithms. For the regularization based adap- 
tive algorithms, the cost function is defined by combining the 
the Ip norm of the error signal with the Iq norm penalty of the 
coefficient vector. 

Regularization plays a fundamental role in adaptive fil- 
tering, however the better performance is not obtained if the 
regularization parameter r is not choosing properly. Accord- 
ing to ([Tol l and (fTST i, the parameter r denotes the importance 
of the li norm term or the intensity of attraction. So a large r 
results in a faster convergence since the intensity of attraction 
increases as r increases. However, steady-state misalignment 
increases as r increases. Therefore, the parameter r are deter- 
mined by the trade-off between adaptation speed and adapta- 
tion quality in particular applications. 

One possible way to find the optimal regularization pa- 
rameter a for four LMS-type algorithms is given in jS]. The 
performance of the regularization based algorithms may be 
improved by using a adaptive regularization factor To verify 
the performance of the LAD type algorithms, a fixed regular- 
ization factor is used. 

We introduce the parameter e > in ( fT2] l in order to pro- 
vide stability and to ensure that a zero-valued component in 
w(n) does not strictly prohibit a nonzero estimate at the next 
step. As empirically demonstrated in Section IV part A, the 



RZA-LAD algorithm is robust to the choice of e. As men- 
tioned in|4|,eshouldbeset slightly smaller than the expected 
nonzero magnitudes of w(n). 

4. SIMULATIONS 

In this section, the performance of the proposed method is as- 
sessed via computer simulations. For comparison purposes, 
we also implement the LMS, RZA-LMS and ZA-LMS algo- 
rithms. Mean square deviation (MSD) is taken as a metric, 
which is defined as 

1 ^ 

^iSD^-Y^W-^'^''^ -^o\\\. (17) 
fc=i 

where w*^*^^ is the coefficient estimated in the fc th independent 
tries and Wo is the real coefficient of the system. 

The noise is impulsive with a a-stable distribution. The 
characteristic exponent a = 1.2, the symmetry parameter 
(3 — 0, the location parameter 7 = and the dispersion (also 
called the scale) 6 — 1. All the simulation results are ob- 
tained by running 500 times independently. The parameters 
are given in Table [T] 

Table 1. Simulation parameters 



Algorithms /ire 



LAD 


5 


X 


10- 


-3 












ZA-LAD 


5 


X 


10- 


-3 


1.5 


X 


10- 


-4 




RZA-LAD 


5 


X 


10- 


-3 


1.5 


X 


10- 


-3 


1 X 10^2 


LMS 


5 


X 


10- 


'3 












ZA-LMS 


5 


X 


10- 


-3 


1.5 


X 


10- 


-4 




RZA-LMS 


5 


X 


10- 


-3 


1.5 


X 


10- 


-3 


1 X 10-2 



To verify the robustness of parameter e, the following ex- 
periment is considered. A linear time invariant system with 
16 coefficients is considered. Initially, we set the 5th tap with 
value 1 and the others to zero, making the system have a spar- 
sity of 1/16. The performance of LAD, ZA-LAD and RZA- 
LAD with different e is compared when the input signal is 
a white Gaussian random sequence with variance of 1 . The 
average MSD is shown in FiglT] 

From Figdl we can see that, the performance of ZA-LAD 
and RZA-LAD is similar when e = 0.001 is used. It is be- 
cause for a small e, e |w3(n)| « 0. The update functions for 
ZA-LAD ([Toll and RZA-LAD are equivalent, then the 
RZA-LAD algorithm reduces to the conventional ZA-LAD 
algorithm. Compared with LAD and ZA-LAD, better steady- 
state performance is obtained for the RZA-LAD algorithm 
when e = 0.01, 0.1, 1, 10, respectively. Another observation 
is that, the convergence rate of RZA-LAD is slower when a 
larger e is used. It is because, e is in the denominator part of 
the third term of the RZA-LAD update function ( fT6l l. A larger 
£ will make the step-size in the third term of (fT6] l becoming 
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Fig. 1. Robustness of parameter e. Gaussian input signal and 
impulsive noise with a stable distribution. 

smaller. The convergence rate is slower due to the relatively 
smaller step-size. 

In the second experiment, similar to the simulation setup 
used in |3 1. A linear time varying system with 16 coefficients 
is considered. Initially, we set the 5th tap with value 1 and 
the others to zero, making the system have a sparsity of 1/16. 
After 3000 iterations, all the odd taps are set to 1, while all 
the even taps remains to be zero, i.e., a sparsity of 8/16. After 
6000 iterations all the even taps are set with value - 1 while all 
the odd taps are maintained to be 1, leaving a completely non- 
sparse system. White Gaussian input signal and impulsive 
noise with a stable distribution. The average MSD is shown 
in FigE] 




Iterations 

Fig. 2. Gaussian input signal and impulsive noise with a sta- 
ble distribution. 

From Fig|2] we can see that, when the system is very 



sparse (before the 3000th iterations), RZA-LAD yields 
fastest convergence rate and best steady-state performance 
compared with LAD and ZA-LAD. And the MSD of the 
LAD and ZA-LAD algorithms are similar. After the 3000th 
iteration, as the number of non-zero efficients increases to 
eight, a similar performance is obtained for LAD and ZA- 
LAD, while the RZA-LAD algorithm maintains the best 
performance among the three algorithms. After 6000th iter- 
ations, the system is completely non-sparse, the performance 
of all the three algorithms are similar. Since the impulsive 
noise is considered, all the three LMS type algorithms are not 
converged. 

The system in the third experiment is the same as the 
second one, except the switching times are set to the 20000 
th iteration and 40000 th iteration, respectively. The input 
signal x{n) is now a correlated signal generated by x{n) = 
0.8x{n — 1) + u{n), where u{n) is a white Gaussian noise. 
The average MSD is shown in Fig[3l 
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Fig. 3. Correlated input signal and impulsive noise with a 
stable distribution. 

From FiglD we can see that, when the system is very 
sparse (before the 20000ift- iterations), both ZA-LAD and 
RZA-LAD yield faster convergence rate and better steady- 
state performance than LAD. And the MSD of the RZA- 
LAD algorithm is lower than the ZA-LAD algorithm. After 
the 20000i/i iteration, as the number of non-zero efficients 
increases to eight, the performance of the ZA-LAD dete- 
riorated, a similar performance is obtained for LAD and 
ZA-LAD, while the RZA-LAD algorithm maintains the best 
performance among the three algorithms. After 40000i/i iter- 
ations, the system is completely non-sparse, the performance 
of the LAD and RZA-LAD algorithms are similar The con- 
vergence rate of the RZA-LAD algorithm is low compared 
with LAD. All the three LMS type algorithms are still not 
converged due to the impulsive noise. 



5. CONCLUSION 

In this paper, the performance of LAD, ZA-LAD and RZA- 
LAD are evaluated for linear time varying system identifica- 
tion under the Gaussian and non-Gaussian (a-stable) noise 
environments. Better performance is obtained for the ZA- 
LAD type algorithms by exploiting the sparse nature of the 
system. RZA-LAD performs best among all the three algo- 
rithms, even when the system is non-sparse. Furthermore, 
the ZA-LAD type algorithms are robust to the non-Gaussian 
noise with a-stable distribution. 

There are two potential future works, the first one is evalu- 
ating the performance of RZA-LAD algorithm using variable 
step size and automatically adapted regularization parameters 
to ensure reliable performance across a broad array of sig- 
nals. The second one is extending the RZA-LAD algorithm 
to nonlinear system such as nonlinear system identification 
using Volterra filter. 
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